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completion of this research. 

The following scientists and other personnel assisted in compiling this report 
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J. M. Disler and W. F. Palmer of Lockheed Engineering and Management Services 
Company, Inc., provided technical inputs and consultation; L. C. Wade of the 
National Aeronautics and Space Administration, Lyndon B. Johnson Space Center, 
reviewed the material for content and provided technical suggestions for 
improvement. 
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1 . INTRODUCTION 


1-1 background 

1 

This report describes how the TY Cams Detailed Analysis Procedures have been 
implemented in conjunction with techniques practiced by CAMS Operations (Ops) 
Verification personnel. This procedural approach, referred to as the "Inte- 
grated Procedure," ms initially implemented by the CAMS Operations Verifica- 
tion personnel during the Transition Year. The procedure consists of three 
major functional activities: labeling as described in the Detailed Analysis 

Procedures, a signature/label review procesr and a segment proportion estimation 
evaluation. The latter function consisted, primarily, of trend analysis 
activities. These functions correspond to the TY CAMS operational activities 
of analyst labeling, quality assurance (QA) and operational verification, 
respectively. 

The implementation of the Integrated Procedure included regionalized segment 
processing, with no specific time constraint for labeling an individual 
segment. Other elements included team labeling at the dot level and use of 
past years ground truth from blind sites and intensive test sites for labeling 
the current data. Additional important inputs into the segment processing 
were use of full frames, crop condition derived from the USDA weekly weather 
and crop reports and historical county crop proportion statistics where 
available. 

It is important to recognize that the implementation approach for the three 
major functions and use of the various input data, including segment regionali- 
zation, varied depending upon data availability, the crop(s) of interest and 
the region involved. 

1-2 DEVELOPMENT OF THE INTEGRATED PROCEDURE 
1.2.1 OPS VERIFICATION IMPLEMENTATION 

The Integrated Procedure was developed through an evolutionary process which 
began during LACIE Phase III with the implementation of the CAMS Operations 
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Verification Function. The verification function was initially used during 
the processing of U.S, Great Plains segments. The function was designed to 
monitor and evaluate the output, both labels and classification results, from 
the Classification and Mensuration Subsystem (CAMS). 

The group performing this function consisted of senior LACIE analysts. The 
verification approach was to (1) re-ai-, • lyze a large sample of the LACIE 
segments and (2) review grouped segments for signature labeling consistency. 

In addition, the verification grouo spent considerable time evaluating and 
applying information gained through the analysis of past years ground truth, 
current meteorological data and cropping practices. It was felt that senior 
analysts working as a team using this information would provide insight into 
effects from environmental conditions such as excessive rainfall or drought. 
Conditions such as these had contributed to mislabeling by operations 
analysts during the early phases of LACIE. 

1.2.2 TREND ANALYSIS IMPLEMENTATION 

During the latter part of Phase III and in the LACIE Transition Year, trend 
analysis techniques were included as an integral part of the detailed segment 
verifications. The result was a labeling-verification technique which inte- 
grated segment-level labeling, a signature-y’eview process and trend analysis 
into a single procedure. The specific techniques employed, for a procedural 
component (i.e., labeling, QA, etc.) varied as previously mentioned, 

1.2.3 INTEGRATION OF LABELING, QA, AND VERIFICATION FUNCTIONS 

An informal procedure which integrated the three operational functions (labeling, 
QA, and verification) was developed for use by the Operations Verification 
Group during the Transition Year processing of the U.S. Great Plains (USGP). 

This "Integrated Procedure" was developed to assist in the verification of 
analyst labeling. Qualitative evaluations of the results produced using this 
procedure indicated that improved labeling accuracy was possible. 
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1.2,4 INTEGRATED PROCEDURE TEST IN THE TY U.S. NORTHERN GREAT PLAINS 


A limitsd test of the "Integrated Procedure" was conducted during the 

2 

Transition Year using the USNGP blind sites, A major objective of this 
test was to determine how well the "Direct Wheat/Barley Separation Procedure" 
would perform in an "integrated procedure" environment, A sub-objective was 
to test and evaluate the complete integrated procedure. In order to 
implement the procedure it was further developed to permit its operation 
with as few as two or three analysts performing joint labeling and discrepancy 
resolution. 

The USNGP test resulted in improved spring small grains labeling accuracy. 

The improved accuracies have been attributed, primarily, to labeling using 
the integrated procedure approach. 

1.2.5 INTEGRATED PROCEDURE TEST IN TY CANADA 

Due to the improved labeling accuracy resulting from the USNGP experiment, 
along with the resource savings that would be realized using an integrated 
operations procedure, it was decided to process the TY Canadian test sites 
using the Integrated Procedure. The evaluation results from this experi- 
ment indicated good labeling accuracy for spring small grains,^ 

1.3 MAXIMAL ANALYSIS LABELING PROCEDURE 

Various aspects of the Integrated Procedure are similar to the Accuracy 
Assessment experimental procedure referred to as the Maximal Analysis Labeling 
Procedure (MALP) . Due to these similarities it has become necessary to 
describe their substantive differences. A comparison of the MALP and the 
Integrated Procedure is presented in Appendix A of this report. 
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2. ELEMENTS OF THE INTEGRATED PROCEDURE 


The Important procedural steps which occur during the three major functional 
activities (labeling, QA, verification) are described and 
specific examples of how Operations Verification personnel implemented the 
Detailed Analysis Procedures are given. Activities or functions incorporated 
into the Integrated Procedure which ?re documented in the Detailed Analysis 
Procedures are referenced. A functional flow depicting the generalized 
functions of the Integrated Procedure and their relation to TY CAMS Operations 
is shown in figure 1. 

Descriptions and discussions regarding each of the three major functions in 
the Integrated Procedure are presented in sections 2.1, 2.2, and 2.3. Again 
it should be noted that procedural variations may be necessary where differences 
in regions, acquisition histories and ancillary data dictate different modes 
of analysis, signature reviewing and trend analyses. Variations of the 
generalized approach described in this document were utilized by the Ops 
Verification Group while processing the TY Canadian data and in performing 
the TY U.S. Northern Great Plains test. 

2.1 STEP I - LABELING 

2.1.1 SEGMENT REGIONALIZATION 

During the implementation of the Integrated Procedure segments were regionalized, 
primarily, by Refined Strata (RS) or Agrophysical Units (APU), although 
occasionally it became necessary to group certain segments using different 
criteria. An episodic event such as drought is an example of a condition 
which resulted in a variation of the regionalization scheme. 

Segment regionalization was included in the procedure because it reduced the 
time required to label and verify signatures. Since similar-appearing segments 
were grouped for analysis the analyst start-up and preparatory times were 
reduced considerably. Analysis of grouped or regionalized segments also 
minimized the confusion that may have resulted from insufficient acquisitions. 

The analyst is more likely to recognize and correctly label signatures which 
have been "recently" viewed on similar or nearby segments. 
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2.1.2 LABELING INPUTS AND PREPARATORY TASKS 

Many of the labeling procedures which are an Integral part of the Integrated 
Procedure are documented in the TY CAMS Detailed Analysis Procedures. These 
procedures include the following analysis-related tasks: 

0 Area Familiarization (section 3,0) 

0 Team Interpretation (section 3.0) 

0 Use of Full Frames (sections 3.3 and 3.6» Part C) 

0 Historical Statistics Usage (sections 3.0 and 3,3, Part C) 

0 Evaluating Past Year's Segment Imagery (section 3.6, Part C) 

0 Spectral Aids Usage (section 6.3.3) 

0 Use of Map Data (section 2.4.1; 

Additional tasks performed by the Ops Verification team while Implementing 
the Integrated Procedure were: 

0 Evaluation of "Weekly Weather and Crop Reports" 

Optimal inputs into segment-level analysis could be best provided by 
agronomists and weather analysts. When this support was not available 
during the operational processing of segments, the analysts interpreted 
the agronomic and weather inputs as described in the Detailed Analysis 
Procedures . 

0 Studying Past Year's Ground Truth Data 

2.1.3 LABELING APPROACH 

Ops Verification personnel participated as a team not only in the general 
signature identification phase (Team Interpretation Task) but also in labeling 
each dot or field in the segment. This was accomplished through independent 
analyses by two analysts which concluded with joint resolution of label 
discrepancies. This particular labeling approach was important because many 
dots were border or edge and the "best" label could be determined only through 
team discussions. 
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2.1.4 SUMMARY OF INTEGRATED PROCEDURE DIFFERENCES FROi‘1 TY CAMS OPERATIONS 

The major differences between the Ops Verification integrated analysis 
labeling approach and the CAMS Operations labeling approach implemented 
during TY can be summarized by restating the additional functions performed 
by the Ops Verification Group: 

a. Regionalized segment processing 

b. Team labeling at the pixel level with joint discrepancy resolution 

c. Use of past years ground truth (this is in addition to the blind site 
segments documented in the Analyst Interpretation Keys, Volumes I and 11)^ 

d. Removal of the processing time constraint on individual segments 

e. Emphasis placed on usage of full frames for the replacement of 5 by 6 nm 
Segment acquisitions not acquired (Operations processing time constraints 
often precluded adequate use of the full frame.) 

^•2 STEP II ~ SIGNATURE/LABEL REVIEW 

Following the labeling of all segments in a predefined region the segments 
were systematically reviewed by the team for consistency in signature label- 
ing. This step was also important in recognizing those segments which were 
mislabeled due to incomplete acquisition histories, For example, this step 
may reveal that a missing key acquisition for a particular segment has 
resulted in small grains being confused with a non-small grain. By comparing 
the signatures within this segment to nearby segments which have "complete" 
acquisition histories the confusion may be recognized and rectified. 

Early or late planted spring small grains omitted during Step I may also be 
detected through this multiple-segment review process. It should be noted 
that potential or actual confusion detected during this step can result in 
Step I being repeated and may involve the rework of more than one 
segment. 

This function closely parallels the CAMS Operations Quality Assurance (QA) 
function described in the Detailed Analysis Procedures, section 1.0. 
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2.3 STEP III - TREND ANALYSES 


Trend analyses or estimate evaluations were used to diagnose the potential 
for occurrence of labeling errors on an Individual segment or groups of 
segments. One of the major objectives of this activity was to not only 
detect labeling errors but to explain all differences betv/een the segments 
being evaluated and the reference data to which they were being compared. 

The reference, data used for comparative studies Included past year's ground 
truth and historical county crop proportion statistics. 

Iitdivldual estimates were evaluated by comparing directly to the analyst 
estimates from previous years or to past year's ground truth proportions, 

If they were available. Multiple estimates v/ere evaluated through trend 
analyses by grouping segments into well defined regions such as Refined 
Strata, CRD's or even entire states. In the case where segments Hh1n a 
CRD or a state were being evaluated the historical crop proportions v/ere used 
as the comparison standard. Evaluations of rrultiple estimates within a 
Refined Strata consisted, primarily, of inspection for estimates which 
appeared to be outliers, either higher or lower than the majority of the 
remaining segments within the Refined Strata, 

Descriptions of the trend analysis approaches are expanded in the following 
sections, 

2*3.1 SINGLE-SEGMENT EVALUATION 

Individual segments wer© evaluated by comparing to past year's analyst estimates 
or ground truth proportions anj attempting to determine if normal year-to- 
year variability was exceeded by any segment. For example, if the proportion 
estimate dropped from 40 percent spring small grains to 20 percent the follow- 
ing year, this would be ''.nsidered a significant change and warrant further 
investigation. Several plausible explanations are: (!) this is a wheat- 

fallow rotation area where the proportion of spring small grains may fluctuate 
from year to year, (2) the cropping practices have changed suddenly due to 
economic factors such as a major shift to another crop, or (3) labeling omission 
errors have occurred. ‘If the latter was suspected, then the segment was 
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re-analyzed with emphasis placed upon evaluating input data such as completeness 
of the reference imagery (AI keys, past ground truth, etc.), meteorological 
data, weekly crop reports, reports on economic factors affecting agricultural 
practices and completeness of the Landsat acquisitions. 

If a significant increase in the proportion estimate had occurred and the 
only plausible explanation for the increase was analyst commission errors, 
then the labels/signatures were re-evaluated for non-small grains confusion. 

An example would be commission of alfalfa to the winter wheat category in 
eastern South Dakota. In this region, alfalfa appears similar to wheat and 
can easily be misidentified as small grains. 

This technique also permits the detection of "new" signatures which have not 
been documented in the TY CAMS Detailed Analysis Procedures (including the 
AI keys), An example of undocumented signatures would be drought-affected 
small grains responses in a region where previous drought has not occurred 
since the collection of Landsat data has commenced, 

2.3.2 REGIONAL TREND ANALYSES 

2.3.2. 1 Historical Crop Proportions 

One method for flagging potential problems on a regional basis was to conduct 
a trend analysis using all segments in a CRD o'r state. Segment estimates, 
for example, for a CRD were compared to historical statistics (using one or 
mbre years historical data) in an attempt to detect a significant change in 
crop acreage. Individual segments were re-analyzed if large changes were 
detected. Frequently, the changes were not due to labeling errors but to 
occurrences such as sampling error or governmental inducements to increase 
or decrease acreage for a particular crop. Another common reason and perhaps 
the most prevalent reason for differences occurring between segment estimates 
and historical statistics was where predominately non-agri cultural areas were 
sampled. In this case a segment with a small grain proportion estimate of 
5 percent may have been located in an area where estimates averaged 30 percent 
per segment. 


2. 3. 2. 2 Refined Strata 


Segment estimates v/ithin Refined Strata or even Agrophysical Units 
were evaluated for outliers. An outlier being an estimate significantly 
different from the remaining segments in the Refined Strata. This procedure 
is particularly useful in areas where reliable historical statistics are 
not available. 

An example of this flagging technique is a 3 percent estimate in a Refined 
Strata where the segment estimates range from 15 to 40 percent. The explanation 
for this large of a difference could be mislabeling but it could also be due 
to the segment being largely non-agricultural . 

2 . 3 . 2 . 3 Aggregated Area Estimates 

The technique of comparing aggregated area estimates to state-level historical 
crop proportions was not employed in the TY Canadian processing or the U.S. 
Northern Great Plains test but it remains a viable trend analysis tool. This 
approach is extremely useful for detecting small labeling errors (0.5 to 3,0 
percent) which have occurred on a large number of segments. Labeling errors 
of this magnitude are generally difficult to detect at the segment level, 
particularly, if the problem is confusion due to missing acquisitions. 

If differences are noted, again the approach is to methodically review Step I 
in an attempt to explain the cause. 


3. SUMMARY AND CONCLUSIONS 


The Integrated Procedure was tested in both the U.S. Northern Great Plains 
and in Canada during the Transition Year and results show that the procedure 
can provide labeling accuracies which exceed those achieved operationally. 

The procedure combines the "TY CAMS Detailed Analysis Procedure" with 
techniques employed by the Operations Verification Group. 

The Integrated Procedure has three basic functional components. These are 
the Team Labeling (Step I), Signature/Label Review (Step II), and Regional 
Trend Analyses (Step III). 

Step I is the implementation of the "TY CAMS Detailed Analysis Procedure" 
(including team labeling), regionalized segment processing, use of past year's 
ground truth, and no specific time constraint for labeling an individual 
segment. Step II is essentially the same as the Operations Quality Assurance 
(QA) function whereby segments are evaluated for labeling consistency. Step 
III consists, primarily, of proportion estimate trend analyses that are 
designed to flag estimates which are outliers. 

It should be noted that Step III (trend analyses) might conceivably introduce 
a bias into the segment estimation process but past experience has shown 
that the capability for flagging labeling errors outweighs any potential 
bias that may result from the trend analyses. 
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APPENDIX A 

COMPARISON OF INTEGRATED PROCEDURE 
AND THE MAXIMAL ANALYSIS LABELING PROCEDURE 


The Maximal Analysis Labeling Procedure (MALP) and the Integrated Procedure 
both incorporate integral components of the Operational Detailed Analysis 
Procedures. Differences in the two procedures are in utilization, segment 
requirements, machine processing, and area estimation requirements . 

Maximal Analysis Labeling Procedure 

The MALP is an experiment designed to produce the most accurate crop labels 
possible from optimal segments selected using very stringent criteria. 
Accuracy Assessment has proposed using MALP for extension of error analysis 
to foreign indicator regions where ground observations are not available. 
Segment pr,’'portion estimates are not the emphasis of this procedure. 

The current MALP processes integrates several labeling products (LIST, 

Badhwar , etc.) with independently derived analysts' labels into final 
consensus labeling decisions. 

Agronomists and meteorologists provide technical support during the segment 
processing that would not normally be feasible in an operational environment. 

Integrated Procedure 

The integrated procedure is an operational procedure that was tested in both 
the U.S. Northern Great Plains and in Canada during the Transition Year. 

It is a combination of the "TY CAMS Detailed Analysis Procedure" and tech- 
niques employed by the Operations Verification Group. The objective is to 
provide labeling accuracies which support unbiased proportion estimation 
using Procedure 1. 


An important technique is the regionalization of segments \\?here segments with 
"complete" acquisition histories are used to aid in the identification of 
confused signatures on segments with incomplete Landsat coverage. Estimate 
evaluations (or Trend Analysis) are used to diagnose potential errors by 
evaluation Of multiple estimates for outliers, either higher or lower than 
the majority of the segments within a Refined Strata. Past year's analyst 
estimates or ground truth proportions are evaluated to determine if normal 
year-to-year variability is exceeded by any segment. Segment estimates 
for a region are compared to historical statistics to detect significant 
changes in crop acreages. Aggregated area estimates are compared to 
state-level historical crop proportions for detection of omitted signatures 
which have occured on a large number of segments. 

The integrated procedure uses independent and team labeling and, like MALP, 
has no specified time constraints. Consensus labeling is not a requirement. 


